Location determination using generalized fingerprinting

ABSTRACT

An RF fingerprinting methodology is generalized to include non-RF related factors. For each fingerprinted tile, there is an associated distance function between two fingerprints (the training fingerprint and the test fingerprint) from within that tile which may be a linear or non-linear combination of the deltas between multiple factors of the two fingerprints. The distance function for each tile is derived from a training dataset corresponding to that specific tile, and optimized to minimize the total difference between real distances and predicted distances. Upon receipt of an inference request, a result is derived from a combination of the fingerprints from the training dataset having the least distance per application of the distance function. Likely error for the tile is also determined to ascertain whether to rely on other location methods.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. patent application Ser. No.13/183,464, filed on Jul. 15, 2011, the contents of which are herebyincorporated by reference in their entirety.

BACKGROUND

Wi-Fi positioning system (WPS) can provide position in certainsituations (such as indoors) by taking advantage of the rapid growth ofwireless access points (WAPs) in urban areas. A provider of this type ofservice maintains a public database and can determine the position for adevice based on the specific access points accessible from the device ineach specific location. The localization technique used for positioningwith wireless access points is based on measuring the intensity of thereceived signal (Received Signal Strength or “RSS”) to more uniquelyidentify each location (usually arranged in a grid comprising aplurality of tiles) using RF fingerprint locating methodologies(hereinafter referred to as “fingerprinting”). Naturally, the accuracyof such approaches depends on the number of positions that have beenentered into the database. The possible signal fluctuations that mayoccur, however, can increase errors and inaccuracies in the path of theuser. To minimize fluctuations in the received signal, certaintechniques can be applied to filter this kind of “noise.”

However, in practical applications, conventional fingerprintingapproaches are difficult to scale and implement. For example,conventional approaches rarely make effective use of crowd-sourced-onlydata and thus often require labor-intensive calibration in the localenvironment. Moreover, such approaches generally do not make of use ofnon-RF related information that may help improve performance, thushindering the use of better alternative location methods becauseaccuracy of conventional fingerprinting is difficult to evaluate withoutusing external data.

SUMMARY

A fingerprinting methodology is generalized to include non-RF relatedfactors such as, for example, GPS (global positioning system) quality(such as HEPE (horizontal estimated position error) information), devicetype information, client identification data, speed of the device, theoperating system (OS) utilized by the device, and so forth. For eachtile, there is an associated distance function between two fingerprints(the training fingerprint and the test fingerprint) from within thattile which may be a linear or non-linear combination of the deltasbetween multiple factors of the two fingerprints. For severalimplementations, the characteristics of the tile may include non-RFinformation such as, for example, a distribution of GPS HEPE.

The distance function for each tile is derived from a training datasetcorresponding to that specific tile and, in certain implementations, toone or more neighboring tiles. This distance function is then optimizedto minimize the total difference between real distances and predicteddistances. Thereafter, upon receipt of an inference request, a resultmay be derived from a combination of the k fingerprints from thetraining dataset having the least distance per application of thedistance function. A test dataset may also be used to predict the likelyerror for the tile, which in turn can be used to ascertain whether thereis a need to rely on other location method(s). Separately, the testdataset can also be used to determine whether cache data forfingerprints should be created and delivered to mobile devices.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

To facilitate an understanding of and for the purpose of illustratingthe present disclosure and various implementations, exemplary featuresand implementations are disclosed in, and are better understood whenread in conjunction with, the accompanying drawings—it being understood,however, that the present disclosure is not limited to the specificmethods, precise arrangements, and instrumentalities disclosed. Similarreference characters denote similar elements throughout the severalviews. In the drawings:

FIG. 1 is an illustration of an exemplary mobile communications network;

FIG. 2A is an exemplary block diagram illustrating a locatingexperimentation framework for analyzing location determination methodsusing location observations divided into a training dataset and a testdataset;

FIG. 2B is an exemplary block diagram illustrating a computing devicefor analyzing modeling algorithms and location inference algorithmsbased on the results of the locating experimentation framework of FIG.2A;

FIG. 3 is an exemplary flowchart illustrating operation of a computingdevice to calculate aggregate accuracy values associated withperformance of location determination methods;

FIG. 4 is an exemplary block diagram illustrating a pipeline forperforming analytics on location determination methods using datasetsderived from location observations;

FIG. 5 is an exemplary experiment process flow diagram illustratingcomparison of the performance of two experiments using differentlocation determination methods;

FIG. 6 is an exemplary block diagram illustrating an experiment group ofthree experiments for generating comparative analytics;

FIG. 7 is an exemplary flowchart illustrating operation of a computingdevice using generalized fingerprints (to include non-RF relatedfactors) with regard to variation location determination methods; and

FIG. 8 shows an exemplary computing environment.

DETAILED DESCRIPTION

The Global Positioning System (GPS) is a satellite navigation systemthat uses more than two dozen GPS satellites that orbit the Earth andtransmit radio signals which are received by and allow GPS receivers todetermine their own location, speed, and direction. Thus, the GPSsatellites transmit signals to GPS receivers on the ground, and the GPSreceivers passively receive these satellite signals and process them(but generally do not transmit any signals of their own).

The horizontal estimated position error (HEPE) is a measure of the GPSreceiver's accuracy with regard to its determination of its location onthe ground (longitude and latitude). For example, if a GPS receiver'sHEPE is 43 feet, the GPS receiver has determined that its calculatedposition (without regard to altitude) is accurate to within 43 feet.Similarly, an estimated position error (EPE) is a measure of the GPSreceiver's accuracy with regard to its determination of itsthree-dimensional location (longitude, latitude, and altitude); however,there is inherent difficulty in calculating altitude with GPS, and thusEPE is generally larger (sometimes substantially larger) than the HEPE.Viewed differently, a HEPE is basically an EPE without the inaccuracy ofan altitude determination.

In general, a GPS receiver requires an unobstructed view of a minimumnumber of GPS satellites in the sky in order to perform a locationdetermination (at least three satellites for longitude and latitude, andat least four satellites to further include altitude). Consequently, GPSreceivers often do not perform well in forested areas, among tallbuildings in a city setting, or inside buildings and other structures.To assist the GPS receiver in such environments, some location devicesmay use various forms of Positioning Services (PS) to assist the GPSreceiver in determining its location or to independently determine thelocation in lieu of the GPS receiver. For example, A-GPS(“Assisted-GPS”) is a well-known PS technology that uses an assistanceserver to reduce the time needed to determine a location using GPS.

Positioning Services (PS) are a combination of computational servers andground-based “beacons” (such as cell towers, discussed further herein)that provide the ability for a location device to obtain its currentlocation and, in certain implementations, to provide additional servicessuch as identifying nearby points-of-interest such as gas stations,hotels, restaurants, banks, stores, coffee shops, shopping, parking,etc. For example, the Business Mobility Framework (BMF) is a PSinfrastructure that allows server-based PS solutions to request andobtain device location information. PS can also be used to supportEnhanced Local Search (ELS) functionality via the Internet to executelocal search queries to find locations and obtain directions to desireddestinations, both indoors and outdoors.

Advanced Forward Link Trilateration (AFLT) is a method of locationdetermination that utilizes base station triangulation to calculatelocation for a mobile communication device. To determine location, themobile device takes measurements of signals from nearby mobilecommunications base stations (a.k.a., “cell towers”) and reportstime/distance readings back to the communication network which are thenused to triangulate an approximate location of the handset. Similar toGPS, at least three surrounding base stations are required to get aposition fix, although AFLT does not use GPS satellites (and only usescell towers) to determine location. Thus the accuracy of AFLT is limitedto the geometry of the cell towers surrounding the device requestinglocation information—the better the triangulation the more accurate thefix. In any event, AFLT enables location services to work indoors,whereas outdoor location services often use the more accurate GPSsignals when available.

When connecting to a mobile communications network, a mobilecommunications device often receives a “fix” (a generalized locationcorresponding to the nearest cell tower that will service the device)within seconds during the registration process. Often these fixes arethen cached for several minutes and, during this time, any queries madeusing the mobile device will reuse the same generalized locationinformation (the fix) on the assumption the mobile device is still inthe same location absent evidence to the contrary (such as a lostsignal).

Mobile locating refers to services provided by telecommunicationcompanies to approximate the location of a mobile communicationsdevice—and thus are a form of PS. The underlying technology is based onmeasuring power levels and antenna patterns. Since a mobilecommunication device generally communicates wirelessly with the basestation closest to it, and the identity of that base station and itslocation are readily ascertainable, the location of the device can becorrectly presumed to be close to the respective base station. Some basestations employing more advanced location systems might also determinethe sector in which the mobile phone resides (i.e., an approximatedirection away from the base station) as well as estimate the distancefrom the base station. Further approximation and refinement may also beachieved by interpolating signals between the device and neighboringbase stations. Where mobile traffic and density of base stations issufficiently high, the precision of an estimated location may bedetermined to within 50 meters of actual location, whereas areas wherebase stations are distantly located one from another (such as a ruralsetting where many miles may lie between base stations) locations may bedetermined much less precisely.

Similarly, networks of land-based positioning transmitters (such asthose comprising beacons) can enable specialized radio receivers todetermine a two-dimensional position (longitude and latitude) on thesurface of the Earth. Often these systems may be generally less accuratethan any of the Global Navigation Satellite Systems (GNSS) such as GPSlargely because the propagation of their signals is not entirelyrestricted to line-of-sight; however, they remain useful forenvironments unsuitable for GPS—such as underground or in indoorenvironments—and the corresponding receivers often require much lesspower than GNSS systems like GPS.

For example, LORAN-C is a terrestrial navigation system most commonlyused to determine the position of a ship or aircraft. LORAN-C uses lowfrequency radio transmitters that use the time interval between radiosignals received from three or more base stations. Recently, LORAN usehas been in steep decline (with GPS being the primary replacement),although there is some interest in revitalizing LORAN—which operates inthe low frequency portion of the EM spectrum from 90 to 110 kHz—sinceits signals are less susceptible to interference and can penetratebetter into foliage and buildings than GPS signals.

Assisted GPS (A-GPS) is a system which, under certain conditions, canimprove the startup performance (or “time-to-first-fix,” TTFF) of a GPSreceiver. A-GPS is used extensively with GPS-capable cellular phones asits development was accelerated by the U.S. Federal CommunicationsCommission's “E911 Mandate” requiring that the location of a mobilecommunications device be made immediately available to emergency calldispatchers.

While standalone or autonomous GPS devices use only the signals from GPSsatellites, an A-GPS device additionally uses PS network resources tohelp it locate and utilize the GPS satellites both faster and better inpoor signal conditions. For example, in areas of very poor signalconditions (such as in a city), GPS signals may suffer multipathpropagation (e.g., bouncing and reflecting off of buildings) or beweakened by passing through signal obstructions such as atmosphericconditions, walls and roofs, or tree cover. Consequently, when firstpowered on in these conditions, some autonomous GPS navigation devicesmay find it difficult to determine a location due to fragmentary signalreception, thereby rendering such devices unable to function unless anduntil clear signals can be received continuously for an adequate periodof time (which may be several minutes).

An A-GPS device addresses these challenges by using data available fromPS in two regards: satellite acquisition and position calculation. Withregard to the former, PS-provided information might include orbital datafor the GPS satellites that may allow the GPS receiver to lock on to aminimal number of satellites more rapidly. Moreover, the network canprovide precise timing information used to render accurate GPSinformation. In addition, the general location of the device asdetermined by the nearby base stations enables the PS to provideinformation pertaining to local ionospheric conditions and otherconditions that can adversely affect GPS signals. Regarding the latter,a PS “assistance server” generally possesses much higher computationalpower than the mobile device and, thus, can be used to more quicklyperform the calculations used to determine location, and particularlythe extremely difficult and complex calculations that use fragmentaryGPS signals received by the mobile device. Indeed, in several A-GPSdevice implementations (such as those known as “MS-Assisted” A-GPSdevices), the amount of CPU and programming used by the GPS receiver canbe substantially reduced by offloading most of the work onto theassistance server. Conveniently, most A-GPS devices have the option offalling back to standalone or autonomous GPS operations when the network(and the assistance server) is unavailable. In addition, many mobilecommunications devices combine A-GPS and other location servicesincluding Wi-Fi positioning, base station triangulation, and otherpositioning technologies.

Mobile communications device locating also tracks the location of adevice even when the device is in motion. To locate the device, thedevice itself emits at least the roaming signal to contact the nextnearby antenna tower, which is a process that does not use an activecall. Location determination may then be done by multilateration basedon the signal strength to nearby antenna masts.

In order to route calls to a mobile communication device, base stationslisten for a roaming signal sent from the device and then collectivelydetermine which specific station is best able to communicate with thephone (e.g., the closest base station with adequate capacity formanaging the device). As the mobile device changes location, the basestations monitor the signal and the device is handed-off (or “roamed”)from a first station to an adjacent second station as appropriate. Thus,by comparing the relative signal strength from multiple antenna towers,a general location of a phone can be roughly determined. The locationcan be even more precisely determined when a base station's antennapattern supports angular determination and phase discrimination. Indeed,the accuracy of various base station locating techniques varies, with aconnection to a single base station (the location of the base stationcorresponding to a “cell identification” as a surrogate for the devicelocation) being the least accurate, triangulation with multiple basestations being moderately accurate, and certain “Forward Link” timingmethods as being the most accurate. Moreover, the accuracy of thesetechniques (collectively referred to as “network-based”) is dependentboth upon the concentration of the base stations—with urban environmentsachieving the highest possible accuracy—as well as the implementation ofthe most current timing methods.

In contrast to network-based techniques, handset-based locationtechnologies generally use the installation of client software on themobile communication device in order to autonomously determine location.Such techniques then determine the location of the device by computinglocation by cell identification and the signal strengths of the home andneighboring cells (i.e., base stations) which is continuously sent tothe carrier network. In addition, if the device is also equipped withGPS then significantly more precise location information may be sentfrom the handset to the carrier.

Hybrid positioning systems use a combination of network-based andhandset-based technologies for location determination. One example wouldbe some modes of A-GPS, which can both use GPS and network informationto compute the location (although in most A-GPS systems all computationsare done by the handset, and the network is only used to initiallyacquire and use the GPS satellites).

FIG. 1 is an illustration of an exemplary mobile communications network5. The mobile communications network 5 may include a visited network 12,a home network 14, and third party networks 16. The visited network 12may also be referred to as a Visited Public Land Mobile Network (VPLMN),a serving network, a roaming network, etc. Home network 14 may also bereferred to as a Home Public Land Mobile Network (HPLMN). The visitednetwork 12 may be a serving network for a mobile communications and/orcomputer (MCC) device 10 which may be operating in or roaming from itshome network 14. Conversely, the visited network 12 and home network 14may be the same network if the MCC device 10 is not roaming.

The visited network 12 may include one or more base stations at theradio access network (RAN) 20, a Mobile Switching Center (MSC)/VisitorLocation Register (VLR) 30, and other network entities not shown in FIG.1 for simplicity. RAN 20 may be a Global System for MobileCommunications (GSM) network, a Wideband Code Division Multiple Access(WCDMA) network, a General Packet Radio Service (GPRS) access network,wireless fidelity (Wi-Fi) network, 14G/Wi-Max network, a Long TermEvolution (LTE) network, CDMA X network, a High Rate Packet Data (HRPD)network, an Ultra Mobile Broadband (UMB) network, etc. GSM, WCDMA, GPRSand LTE are part of Universal Mobile Telecommunication System (UMTS) andare described in documents from an organization named “3rd GenerationPartnership Project” (3GPP). CDMA X and HRPD are part of cdma2000, andcdma2000 and UMB are described in documents from an organization named“3rd Generation Partnership Project 2” (3GPP2). The MSC may performswitching functions for circuit-switched calls and may also route ShortMessage Service (SMS) messages. The VLR may store registrationinformation for terminals that have registered with visited network 12.

Home network 14 may include a Home Location Register(HLR)/Authentication Center (AC) 40 and other network entities not shownin FIG. 1 for simplicity. The HLR may store subscription information forterminals (including MCC device 10) that have service subscription withhome network 14. The AC may perform authentication for terminals(including MCC device 10) having service subscription with home network14.

Third party networks 16 may include a router or switch 50, a PublicSwitched Telephone Network (PSTN) 70, and possibly other networkentities not shown in FIG. 1. Router or switch 50 may routecommunications between MSC/VLR 30 and a wide area network (WAN) 60 (suchas the Internet). PSTN 70 may provide telephone services forconventional wireline telephones, such as a telephone 80. Of course,FIG. 1 shows only some of the network entities that may be present inthe visited network 12 and the home network 14. For example, visitednetwork 12 may include network entities supporting packet-switched callsand other services, as well a location server to assist in obtaininglocation information for a terminal, e.g., MCC device 10, as discussedelsewhere herein.

The MCC device 10, as a wireless communications terminal, may be also bethought of (and variously referred to as) a mobile station (MS) in GSMand CDMA X, a user equipment (UE) in WCDMA and LTE, an access terminal(AT) in HRPD, a SUPL enabled terminal (SET) in Secure User PlaneLocation (SUPL), a subscriber unit, a station, and so forth. The MCCdevice 10 may also comprise or communicate with a personal navigationdevice (PND), and satellite signal reception, assistance data reception,and/or position-related processing may occurs at the MCC device 10 or,alternately, at the PND. The MCC device 10 may have a servicesubscription with home network 14 and may be roaming in visited network12, as shown in FIG. 1.

When activated, the MCC device 10 may receive signals from RAN 20 invisited network 12 and communicate with the RAN 20 to obtaincommunication services. The MCC device 10 may also communicate with homenetwork 14 for communication services when not roaming. The MCC device10 may also receive, via its PND, signals from one or more satellites 90which may be part of a satellite positioning system (SPS). As usedherein an SPS may include any combination of one or more global and/orregional navigation satellite systems and/or augmentation systems, andSPS signals may include SPS, SPS-like, and/or other signals associatedwith such one or more SPS. As such, the MCC device 10 may measuresignals from satellites 90 and obtain pseudo-range measurements for thesatellites. The MCC device 10 may also measure signals from basestations in RAN 20 and obtain timing and/or signal strength measurementsfor the base stations. The pseudo-range measurements, timingmeasurements and/or signal strength measurements may be used to derive aposition estimate or location estimate and location information for theMCC device 10, as discussed elsewhere herein.

Wi-Fi positioning system (WPS) can also provide position in certainsituations (such as indoors) by taking advantage of the rapid growth ofwireless access points (WAPs) in urban areas. A provider of this type ofservice maintains a public database and can determine the position for adevice based on the specific access points accessible from the device ineach specific location. The localization technique used for positioningwith wireless access points is based on measuring the intensity of thereceived signal (Received Signal Strength or “RSS”) and “fingerprinting”the possible locations (usually arranged in a grid comprising aplurality of tiles). The accuracy of such approaches depends on thenumber of positions that have been entered into the database. Thepossible signal fluctuations that may occur, however, can increaseerrors and inaccuracies in the path of the user. To minimizefluctuations in the received signal, certain techniques can be appliedto filter this kind of “noise,” and various implementations disclosedherein are related to such techniques.

It should be noted that a difference between fingerprinting andtriangulation lies in the database used by the former with which thepossible locations are compared to. More specifically, infingerprinting, the signal levels detected from a Wi-Fi device are stillfound using multiple access points as in triangulation. However, incontrast to triangulation which attempts to determine a distance fromeach access point, in fingerprinting the combined signals are thencompared with the database that contains power levels of each point onthe map and the fingerprinting database is further calibrated bymeasuring power at different locations. In this way, fingerprinting isable to take into account such factors as reflection, attenuation, andmulti-path signal propagation which can create uncertainty in othertechniques such as triangulation.

However, in practical applications, conventional fingerprintingapproaches are difficult to scale and implement for several reasons. Forexample, conventional approaches rarely make effective use ofcrowd-sourced-only data and thus often use labor-intensive calibrationin the local environment. Moreover, such approaches generally do notmake of use of non-RF related information that may help improveperformance, thus hindering the use of better alternative locationmethods because accuracy of conventional fingerprinting is difficult toevaluate without using external data.

For various implementations disclosed herein, a fingerprintingmethodology is generalized to include non-RF related factors such as,for example, GPS quality (such as HEPE information), device typeinformation, client identification data, speed of the device, theoperating system (OS) utilized by the device, and so forth. For eachtile, there is an associated distance function between two fingerprints(the training fingerprint and the test fingerprint) from within thattile which may be a linear or non-linear combination of the deltasbetween multiple factors of the two fingerprints. For severalimplementations, the characteristics of the tile may include non-RFinformation such as, for example, a distribution of GPS HEPE.

The distance function for each tile is derived from a training datasetcorresponding to that specific tile and, in certain implementations, toone or more neighboring tiles. This distance function is then optimizedto minimize the total difference between real distances and predicteddistances. Thereafter, upon receipt of an inference request, a resultmay be derived from a combination of the k fingerprints from thetraining dataset having the least distance per application of thedistance function. A test dataset may also be used to predict the likelyerror for the tile, which in turn can be utilized to ascertain whetherthere is a need to rely on other location method(s). Separately, thetest dataset can also be used to determine whether cache data forfingerprints should be created and delivered to mobile devices.

FIG. 2A is an exemplary block diagram illustrating a locatingexperimentation framework for analyzing location determination methodsusing location observations (fingerprints obtained from known locations)that are divided into a training dataset and a test dataset. FIG. 2B isan exemplary block diagram illustrating a computing device for analyzingmodeling algorithms and location inference algorithms based on theresults of the locating experimentation framework of FIG. 2A.

Referring to FIGS. 2A and 2B (collectively referred to hereinafter asFIG. 2), various implementations disclosed herein are operable in anenvironment in which MCC devices such as mobile computing devices orother observing computing devices 210 (an example of which is describedwith respect to FIG. 8) observe or detect one or more base stations 212at approximately the same time (e.g., an observation time value 216)while the device is at a particular location (e.g., an observationlocation 214). The set of observed base stations 212, the observationlocation 214, the observation time value 216, and possibly otherattributes constitute a location observation as well as non-RF relatedfactors 100. The mobile computing devices detect or observe the basestations 212, or other cell sites, via one or more radio frequency (RF)sensors associated with the mobile computing devices. Aspects of thedisclosure are operable with any base station supporting any quantityand type of wireless communication modes including cellular divisionmultiple access (CDMA), Global System for Mobile Communication (GSM),wireless fidelity (Wi-Fi), 4G/Wi-Max, and the like. Exemplary basestations 212 include cellular towers (or sectors if directional antennasare employed), base stations, base transceiver stations, base stationsites, wireless fidelity (Wi-Fi) access points, satellites, or otherwireless access points (WAPs). While aspects of the disclosure may bedescribed with reference to base stations 212 implementing protocolssuch as the 802.11 family of protocols, implementations of thedisclosure are operable with any base station for wirelesscommunication. Moreover, while aspects of the disclosure may bedescribed with reference to any specific base station for wirelesscommunication (e.g., “base station”), such implementations explicitlyinclude, for alternative implementations, the use of any other basestation for wireless communication (e.g., “cell tower”), and thus termsreferring to base stations 212 for wireless communication are usedinterchangeably herein without loss of generality.

Referring to FIG. 2A, an exemplary block diagram illustrates thelocation experimentation framework for analyzing location determinationmethods using both RF-based location observations 102 as well as non-RFrelated factors 100 (such as GPS HEPE) which are together grouped into atraining dataset 106 and a test dataset 108. The training dataset 106includes training location observations, and the test dataset 108includes test location observations. The location experimentationframework includes an experimental dataset constructor 104, whichdivides location observations 102 and non-RF related factors 100 intothe training dataset 106 and the test dataset 108. In someimplementations, the training dataset 106 and the test dataset 108 aremutually exclusive (e.g., no overlap). In other implementations, atleast one location observation and at least one non-RF location factorare included in both the training dataset 106 and the test dataset 108.

Referring more generally to FIG. 2 (i.e., both FIGS. 2A and 2B), andusing locating method dependent modeling 112 (e.g., a modeling algorithm228 and a location inference algorithm 230 in FIG. 2B), models 114 areconstructed from the training dataset 106. An inference engine 118applies at least one of the location inference algorithms to the testdataset 108 and uses the models 114 to infer location inference results120 such as device location estimates 224 for the observing computingdevices 210. In some implementations, the inference engine 118 also usesthird-party models 116 to produce the location inference results 120.The device location estimates 224 represent inferred locations of theobserving computing devices 210 in each of the location observations 102in the test dataset 108, taking into account the non-RF related factors100 that are available. Analytics scripts 122 analyze the inferenceresults 120 in view of the training dataset 106 and the test dataset 108to produce analytic report tables 124 and statistics and analyticsstreams 126. The analytics scripts 122, in general, calculate theaccuracy of the locating method, such as an error distance. Thestatistics and analytics streams 126 are used by visualization anddebugging tools 128 and by the inference engine 118.

Referring to FIG. 2B, an exemplary block diagram illustrates a computingdevice 202 for analyzing modeling algorithms 228 and location inferencealgorithms 230. In some implementations, the computing device 202represents a cloud service for implementing aspects of the disclosure.For example, the cloud service may be a location service accessinglocation observations 102 stored in a base station store. In suchimplementations, the computing device 202 is not a single device asillustrated, but rather a collection of a plurality of processingdevices and storage areas arranged to implement the cloud service. Anexample computing device is described with respect to FIG. 8.

In general, the computing device 202 represents any device executinginstructions (e.g., as application programs, operating systemfunctionality, or both) to implement the operations and functionalityassociated with the computing device 202. The computing device 202 mayalso include a mobile computing device or any other portable device. Insome implementations, the mobile computing device includes a mobiletelephone, laptop, tablet, computing pad, netbook, gaming device, and/orportable media player. The computing device 202 may also include lessportable devices such as desktop personal computers, kiosks, andtabletop devices. Additionally, the computing device 202 may represent agroup of processing units or other computing devices.

The computing device 202 has at least one processor 204 and a memoryarea 206. The processor 204 includes any quantity of processing units,and is programmed to execute computer-executable instructions forimplementing aspects of the disclosure. The instructions may beperformed by the processor 204 or by multiple processors executingwithin the computing device 202, or performed by a processor external tothe computing device 202. In some implementations, the processor 204 isprogrammed to execute instructions such as those described elsewhereherein.

The computing device 202 further has one or more computer readable mediasuch as the memory area 206. The memory area 206 includes any quantityof media associated with or accessible by the computing device 202. Thememory area 206 may be internal to the computing device 202 (as shown inFIG. 2B), external to the computing device 202 (not shown), or both (notshown). The memory area 206 stores, among other data, one or morelocation observations 102 such as location observation #1 throughlocation observation #X, as well as any non-RF related factors 100. Inthe example of FIG. 2B, each of the location observations 102 includes aset of one or more base stations 212, an observation location 214, anobservation time value 216, and other properties describing the observedbase stations 212 and/or the observing computing device (which mayinclude the non-RF related factors 100). An exemplary observationlocation 214 may include values for a latitude, longitude, and altitudeof the observing computing device as determined by RF fingerprintingand/or utilizing available non-RF related data. For example, theobservation location 214 of the observing computing device may bedetermined via a global locating system (e.g., GPS) receiver associatedwith the observing computing device.

The computing device 202 may receive the location observations 102 (aswell as any non-RF related factors 100) directly from the observingcomputing devices 210. Alternatively or in addition, the computingdevice 202 may retrieve or otherwise access one or more of the locationobservations 102 (or non-RF related factors 100) from another storagearea such as a base station store. In such implementations, theobserving computing devices 210 transmit, via a network, the locationobservations 102 (and the non-RF related factors 100) to the basestation store for access by the computing device 202 (and possibly otherdevices as well). The base station store may be associated with, forexample, a locating service that crowd-sources the location observations102. The network includes any means for communication between theobserving computing devices 210 and the base station store or thecomputing device 202.

As described herein, aspects of the disclosure operate to divide,separate, construct, assign, or otherwise create the training dataset106 and the test dataset 108 from the location observations 102 and thenon-RF related factors 100 (e.g., non-RF related location factors).Aspects of the disclosure further calculate, using various models, theestimated locations (e.g., device location estimates 224) of theobserving computing devices 210 in the test dataset 108. Each of thedevice location estimates 224 identifies a calculated location of one ofthe observing computing devices 210 (e.g., mobile computing devices) inthe test dataset 108.

The memory area 206 further stores accuracy values 226 derived from acomparison between the device location estimates 224 and thecorresponding observation locations, as described herein. The accuracyvalues 226 represent, for example, an error distance. The memory area206 further stores one or more modeling algorithms 228 and one or morelocation inference algorithms. Alternatively or in addition, themodeling algorithms and location inference algorithms are storedremotely from the computing device 202. Collectively, the modelingalgorithms and location inference algorithms may be associated with oneor more of a plurality of location determination methods, and providedby a locating service.

The memory area 206 further stores one or more computer-executablecomponents. Exemplary components include a constructor component 232, amodeling component 234, an inference component 236, an error component238, a scaling component 240, and a characterization component 242. Theconstructor component 232, when executed by the processor 204, causesthe processor 204 to separate the crowd-sourced location observations102 and the non-RF related location factors into the training dataset106 and the test dataset 108. The constructor component 232 assigns thecrowd-sourced location observations 102 to one or more geographic tilesor other geographic areas based on the observation locations 214 in eachof the crowd-sourced location observations 102. In some implementations,the crowd-sourced location observations 102 (and/or the non-RF relatedlocation factors 100) may be grouped by base station to enable searchingfor location observations 102 based on a particular base station ofinterest.

The modeling component 234, when executed by the processor 204, causesthe processor 204 to determine the location model 222 based on thelocation observations in the training dataset 106. In implementationsthat contemplate base station location estimate, for each base station,the base station location estimates are calculated based on theobservation locations in the training dataset 106 associated with thebase station. That is, aspects of the disclosure infer the location ofeach base station based on the location observations in the trainingdataset 106 that involve the base station. As a result, in suchimplementations, the modeling component 234 generates models 114including a set of base stations 212 and approximate locations of thebase stations 212.

The modeling component 234 implements at least one of the modelingalgorithms 228. The inference component 236, when executed by theprocessor 204, causes the processor 204 to determine, for each of thelocation observations in the test dataset 108, the device locationestimate for the observing computing device 210 based on the appropriatemodel determined by the modeling component 234. The inference component236 implements the location inference algorithms 230, and is operablewith any exemplary algorithm (e.g., refining algorithm) for determininga location of one of the observing computing devices 210 based on thelocation model 222, as known in the art. For each of the locationobservations in the test dataset 108, the inference component 236further compares the device location estimate 224 for the observingcomputing device 210 to the known observation location 214 of theobserving computing device 210 in the test dataset 108 to calculate theaccuracy value 226.

The error component 238, when executed by the processor 204, causes theprocessor 204 to calculate an aggregate accuracy value for each of thetiles based on the calculated accuracy values 226 of the locationobservations assigned thereto in the test dataset 108. For example, theerror component 238 groups the calculated accuracy values 226 of thetest dataset 108 per tile, and calculates the aggregate accuracy valuefor each tile using the grouped accuracy values 226.

The scaling component 240, when executed by the processor 204, causesthe processor 204 to adjust a size of the tiles to analyze the accuracyvalues 226 aggregated by the error component 238. The size correspondsto one of a plurality of levels of spatial resolution. As the size ofthe tiles changes, aspects of the disclosure re-calculate the aggregateaccuracy values, and other analytics, for each of the tiles.

The characterization component 242, when executed by the processor 204,causes the processor 204 to calculate data quality attributes and datadensity attributes for the crowd-sourced location observations 102 inparticular view of the non-RF related factors 100. Exemplary dataquality attributes and exemplary data density attributes are describedbelow with reference to FIG. 4. Further, the error component 238 mayperform a trend analysis on the data quality attributes and the datadensity attributes calculated by the characterization component 242. Thetrend analysis illustrates how these statistics evolve over time. Forexample, for a given tile, the trend analysis shows how fast theobservation density increases or how the error distance changes overtime. In some implementations, the characterization component 242compares the calculated aggregate accuracy values to base stationdensity in, for example, a scatter plot.

Referring next to FIG. 3, an exemplary flowchart illustrates operationof the computing device 202 (e.g., cloud service) to calculate aggregateaccuracy values associated with performance of location determinationmethods. In some implementations, the operations illustrated in FIG. 3are performed by a cloud service such as a location determinationservice. At 302, the training dataset 106 and the test dataset 108 areidentified. For example, the crowd-sourced location observations 102 andthe non-RF related factors 100 (such as non-RF location or positionedobservations) are divided into the training dataset 106 and the testdataset 108. The crowd-sourced location observations 102 may be dividedbased on the observation times associated therewith. For example, thetraining dataset 106 may include the crowd-sourced location observations102 and/or the non-RF related factors 100 that are older than two weeks,while the test dataset 108 may include the crowd-sourced locationobservations 102 and/or the non-RF related factors 100 that are lessthan two weeks old. Aspects of the disclosure contemplate, however, anycriteria for identifying the training dataset 106 and the test dataset108. For example, the location observations 102 (as well as the non-RFrelated factors 100 in certain implementations) may be divided based onone or more of the following: geographic area, type of observingcomputing device, location data quality, mobility of observing computingdevice, received signal strength availability, and scan time difference(e.g., between the ends of Wi-Fi and GPS scans).

Further, in some implementations, the crowd-sourced locationobservations 102 and/or the non-RF related factors 100 are preprocessedto eliminate noisy data or other data with errors. For example, thecrowd-sourced location observations 102 may be validated through datatype and range checking and/or filtered to identify locationobservations 102 that have a low mobility indicator.

Each of the crowd-sourced location observations 102 has an observingcomputing device (e.g., a mobile computing device) associated therewith.At 304, the crowd-sourced location observations 102 are assigned to oneor more geographic areas. The crowd-sourced location observations 102may be assigned based on a correlation between the geographic areas andthe observation locations 214 associated with each of the crowd-sourcedlocation observations 102.

At 306, a model is determined from the training dataset 106, and alocation estimate is calculated by executing a selection of at least oneof the modeling algorithms. In addition to RF related data, thegeneralized fingerprinting or distance determining method mayincorporate additional attributes such as device type and so forth. Asknown and appreciated by skilled artisans, finding the least distance isa classic optimization problem requiring the determination of a functionthat minimizes the error of predicted error distance versus actual errordistance which, for the implementations disclosed herein, pertain to thetraining data set. Regardless, the model built from the training datasetin this approach may be similar to the training dataset itself in thatit might have an observation (i.e., a fingerprint with its associatedGPS location), along with the identification of the specific tile foreach observation (at a determined level of detail within the tilesystem). Hence the inference call from the client provides a fingerprintwhich can be compared against the observations in the model and rankedbased on the distance from the client fingerprint. Many fingerprintproperties may be used such as the number of common base towersdetected, the signal strength of those base towers, the device type,HEPE, speed, and so forth, and lesser distances are given higher scores(i.e., the least distance may be given the highest score). Thetop-ranked observations may then be used to determine user location.

At 308, device location estimates 224 for the observing computingdevices 210 associated with the location observations in the testdataset 108 are determined. For example, the device location estimatefor the observing computing device 210 in one of the locationobservations in the test dataset 108 is determined based on the locationmodel 222. The device location estimates 224 are calculated by executinga selection of at least one of the location inference algorithms 230.

At 310, for each of the location observations in the test dataset 108,the determined device location estimate 224 is compared to theobservation location 214 of the observing computing device 210associated with the location observation. The comparison produces theaccuracy value 226. In some implementations, the accuracy value 226represents an error distance, a distance between the observationlocation 214 of the observing computing device 210 and the calculateddevice location estimate 224 of the observing computing device 210, orany other measure indicating accuracy.

At 312, for each of the geographic areas, the accuracy values 226associated with the location observations assigned to the geographicarea from the test dataset 108 are combined to calculate an aggregateaccuracy value. For example, a mean, median, cumulative distributionfunction, trend analysis, or other mathematical function may be appliedto the accuracy values 226 for each of the geographic areas to producethe aggregate accuracy value for the geographic area.

In some implementations, the training dataset 106 and the test dataset108 are characterized or otherwise analyzed to produce dataset analyticsat 305. Exemplary dataset analytics include data quality attributes,data density attributes, and an environment type (e.g., rural, urban,dense urban, suburban, indoor, outdoor, etc.) for each of the geographicareas. Further, the performance of the selected modeling algorithm 228and the selected location inference algorithm 230 may be analyzed toproduce quality analytics. In some implementations, the datasetanalytics are correlated to the quality analytics to enableidentification and mapping between qualities of the input data to theresulting performance of the location methods.

Another way to implement the location model is to pick a small tile in adensely populated area and determine the least distance values betweenall pairs of observations in that tile. A good least distance functionmay have scores above a certain limit for all pairs of observations(given that they are all in a very small tile). The location model mayalso be tuned based on the different parameters and then individuallyvarying one parameter at a time to gauge its impact. For example, theimpact of two fingerprints not generated from the same device might besignificant compared to the number of common base towers detected bytheir corresponding devices. Moreover, fingerprint based algorithms areexpected give different results based on the different selections oftest and training data sets such that some might work better in verydensely populated urban areas and poorly in more suburban or ruralareas, in which case density might be more important than HEPE (forwhich the location model can adjust for accordingly). Regardless, theleast distance observation may be used in the model as the likelyinferred position for the device, and this can also be varied by themodel in selecting the centroid of a number of model fingerprintpositions (comprising the least distanced fingerprints or thefingerprints that have a least distance score over some threshold).Using the fingerprint, the model may then also perform some form ofcorrection based on, for example, the speed and heading of the device tobetter model the fingerprint position.

Referring next to FIG. 4, an exemplary block diagram illustrates apipeline for performing analytics on location determination methodsusing datasets derived from location observations 102 and non-RF relatedfactors 100. The experimental dataset constructor 104 takescrowd-sourced location observations 102 and the non-RF related factors100 (such as non-RF location or positioned observations) and generatesthe training dataset 106 and the test dataset 108 based on, for example,filter settings at 406. Dataset analytics are generated for the trainingdataset 106 and the test dataset 108 at 410. The dataset analytics arestored as dataset characterizations 412.

Exemplary dataset analytics include characterizations in terms of one ormore of the following, at various levels of spatial resolutions:cumulative distribution function, minimum, maximum, average, median, andmode. The dataset analytics include data quality attributes, datadensity attributes, and environment type. Exemplary data qualityattributes include one or more of the following: HEPE, speed/velocitydistribution, heading distribution, and delta time stamp. The HEPErepresents the estimated 95% location error (e.g., in meters). The deltatime stamp represents the difference (e.g., in milliseconds) between thecompletion of a Wi-Fi access scan and a GPS location fix. Exemplary datadensity attributes include one or more of the following: observationdensity (e.g., the number of observations per square kilometer), basestation density (e.g., the number of base stations 212 per squarekilometer), distribution of the number of base stations 212 per scan,and distribution of observations per base station 212.

Preprocessing, modeling, and inference are performed specific to aparticular locating method. For example, the locating method includes atleast one of the modeling algorithms 228 and at least one of thelocation inference algorithms 230. Models 114 are generated at 414 basedon the training dataset 106. The inference engine 118 uses the models114 at 416 to process the test dataset 108 and produce inference results120.

Experiment analytics 418 are next performed. Analytics on the inferenceresults 120 are aggregated at 420 to generate, for example, a cumulativedistribution function (CDF) per geographic tile. The aggregatedanalytics are stored as inference analytics 422. The inference analyticscombine different inference results 120 together and aggregate them bygeographic tile. The dataset characterization and inference analyticsare aggregated to generate, for example, density to accuracy charts at424. Further, pairwise delta analytics 426 and multi-way comparativeanalytics 428 may also be performed. The pairwise delta analytics 426and the multi-way comparative analytics 428 enable finding a correlationbetween training data properties and error distance analytics reports.The result of this data may be visually analyzed as a scatter graph orpivot chart. For example, the pairwise delta analytics 426 examine thedifference between error distances of two alternative methods versus adata metric such as base station density. In another example, themulti-way comparative analytics 428 illustrate the relative accuracy ofmultiple experiments give a particular data quality or density metric.Other analytics are contemplated, such as per base station analytics.

In some implementations, the experiment analytics 418 have severallevels of granularity. There may be individual inference errordistances, intra-tile statistics (e.g., 95% error distance for a giventile), inter-tile analytics (e.g., an accuracy vs. base station densityscatter plot for an experiment), and inter-experiment comparativeanalytics.

Exemplary intra-tile statistics include one or more of the following:test dataset analytics (e.g., base station total, base station density,base station count per inference request), query success rate,cumulative distribution function (e.g., 25%, 50%, 67%, 90%, and 95%),and other statistics such as minimum, maximum, average, variance, andmode. Exemplary inter-tile analytics are summarized form training dataover a plurality of geographic tiles and may include scatter plotsillustrating one or more of the following: error vs. observationdensity, error vs. observed base station density, error vs. number ofaccess points used in the inference request, and error vs. data densityand data quality.

Aspects of the disclosure may further relate dataset analytics toaccuracy analytics. In some implementations, there is a continuous model(e.g., no estimate of base station location) and a discrete model,although other models are contemplated. In the continuous model, D is adata density function and Q is a data quality function. The function Dis a data density function of observation density, base station density,and the distribution of the number of access points per scan. Thefunction Q is a data quality function of HEPE distribution, speeddistribution, delta time stamp distribution, and heading distribution.For a given training dataset 106 and a particular geographic tile,aspects of the disclosure calculate the data density indicator and thedata quality indicator using the functions D and Q. When combined with aselected accuracy analytic A such as 95% error distance, aspects of thedisclosure operate to create a three-dimensional scatter plot, whereeach data point in the plot is of the form (X=D, Y=Q, Z=A).

In the discrete model, for a particular training dataset 106, aspects ofthe disclosure classify each geographic tile that covers an area of thetraining dataset 106 as (D, Q), where values for D and Q are selectedfrom a discrete set of values (e.g., low, medium, and high). As crowdsourced data grows in volume and improves in quality, more tiles areexpected to move from (D=low, Q=low) to (D=high, Q=high).

Referring next to FIG. 5, an exemplary experiment process flow diagramillustrates comparison of the performance of two experiments usingdifferent location determination methods. The process begins at 502. Thetraining dataset 106 and the test dataset 108 are generated at 504 fromthe crowd-sourced location observations 102 and the non-RF relatedfactors 100. At 506, a first experiment is conducted using a particularlocating method (e.g., using at least one of the modeling algorithms 228and at least one of the location inference algorithms 230 on aparticular training dataset 106 and test dataset 108). Performanceanalytics are generated for the first experiment at 508, as describedherein, and then analyzed at 510. For example, an error distance graphper tile may be created.

At 512, a second experiment is conducted using another locating method(e.g., different modeling algorithm 228 and/or different locationinference algorithm 230 from the first experiment). Performanceanalytics are generated for the second experiment at 514, as describedherein, and then analyzed at 516. Pairwise analytics are generated forthe first and second experiments at 518, and then analyzed at 520. Forexample, an error distance difference per tile may be created for eachof the locating methods to enable identification of the locating methodproviding the better accuracy (e.g., smaller error distance).

At 522, the analyzed analytics data may be reviewed to draw conclusionssuch as whether a correlation can be seen between any of thecharacteristics of the training dataset 106 and error distance, whetherone locating method performs better than another for a particularcombination of data quality and data density, and the like. If anomaliesare detected (e.g., two tiles with similar observation density showvaried error distance), the raw location observation data may bedebugged at 526. Further, the experiments may be re-run after pivotingon a different parameter at 524. For example, if there is no correlationbetween observation density and error distance, the experiments may bere-run to determine whether there is a correlation between HEPE anderror distance. In addition, at 528, the results are recorded and theprocess may end.

In some implementations, the operations illustrated in FIG. 5 maygenerally be described as follows. In a first experiment, a first one ofa plurality of the modeling algorithms 228 is selected and executed withthe training dataset 106 as input. This results in the creation of thelocation model 222 based on the training dataset 106. A first one of aplurality of location inference algorithms 230 is selected and executedwith the test dataset 108 and the location model 222 as input. Thisresults in creation of device location estimates 224 for the observingcomputing devices 210. The device location estimates 224 are compared tothe observation locations 214 of the observing computing devices 210 tocalculate accuracy values 226. The accuracy values 226 are assigned tothe geographic areas based on the observation location 214 of thecorresponding location observations in the test dataset 108. Aggregateaccuracy values are created by combining the accuracy values 226 fromeach of the geographic areas.

In a second experiment, the location model 222 is recalculated using asecond selected modeling algorithm 228 and the device location estimates224 are recalculated using a second selected location inferencealgorithm 230. The aggregate accuracy values are re-calculated for eachof the geographic areas to enable a comparison of the selected modelingalgorithms 228 and the selected location inference algorithms 230between the first experiment and the second experiment.

In some implementations, the computing selects the first or secondmodeling algorithms 228 and/or the first or second location inferencealgorithms 230 as the better-performing algorithm based on a comparisonbetween the aggregated accuracy values of the first experiment and thesecond experiment.

In some implementations, a size of one or more of the geographic areasmay be adjusted. The aggregate accuracy value, or other qualityanalytics, is calculated for each of the re-sized geographic areas byre-combining the corresponding accuracy values 226.

Referring next to FIG. 6, an exemplary block diagram illustrates anexperiment group 602 of three experiments for generating comparativeanalytics. Each of the Experiment A 604, Experiment B 606, andExperiment C 608 represent the application of a selected modelingalgorithm 228 and a selected location inference algorithm 230 to aparticular training dataset 106 and test dataset 108. Datasetconstructor scripts 610 create the training dataset 106 and the testdataset 108 from the location observations 102 and the non-RF relatedfactors 100. Dataset analytic scripts 612 create training datasetcharacteristics 616 and test dataset characteristics 614 at the basestation, tile, and world (e.g., multiple tiles) levels to characterizethe output at multiple levels of spatial resolution. In this way,aspects of the disclosure characterize the input data at multiple levelsof spatial resolution.

Experiment A 604 applies a particular location method 618. This includesexecuting modeling scripts 620 to create models 114. Inference scripts622 apply the models 114 to the test dataset 108 to create the inferenceresults 120. Inference analytics are obtained from the inference results120 to produce accuracy analytics 624 at the base station, tile, andworld (e.g., multiple tiles) levels.

Experiment B 606 and Experiment C 608 are performed using differentlocation methods. Comparative analytic scripts 626 are performed on theaccuracy analytics 624 from Experiment A 604 as well as the output fromExperiment B 606 and Experiment C 608. Multi-way and pairwisecomparative, delta, and correlation analytics are performed at 628.

FIG. 7 is an exemplary flow chart 700 illustrating operation of thecomputing device using generalized fingerprints (to include non-RFrelated factors) with regard to variation location determinationmethods. Referring to FIG. 7, the training and test datasets thatconsist of generalized fingerprints are constructed at 712. At 714, thefingerprints are then partitioned per a mapping tile system. At 716, amodel is created that consists of training data set and a distancefunction associated for each tile. At 718, the list of fingerprints fromthe training dataset that will be used to compute the distance functionis filtered, for example, to include all the fingerprints in thetraining dataset or a subset depending on the characteristics of thetile and the computational complexity.

At 720, each tile is then associated with a distance function, which canbe linear or non-linear and is a function of the deltas between multiplefactors between two fingerprints in the training dataset. This distancefunction may also incorporate some of the factors of the fingerprintthat may not be available during an inference request, like GPS qualityor the speed of the device while traveling in a vehicle, for example. Inany event, the objective is to find the distance function that minimizesthe differences between actual distances and predicted distances. Forcertain implementations, as the training dataset grows the distancefunction may be updated incrementally, and while a linear function maybe less complex to solve, a non-linear function might be used inalternative implementations.

At 722, and given an inference request (IR), a list of fingerprints isselected from the training dataset against which the distance functionwas computed. Then, at 724, the k (a predetermined threshold number)nearest fingerprints in the training dataset are found (or determined)based on the distance function for the corresponding tile, where thepredicted location is a combination of locations of the k-nearestfingerprints. At 726, an error distance curve (representative of anaccuracy characteristic) for the tile is determined based on thedistance function and test data for that tile. If the accuracy isadequate at 728, then at 730 the fingerprinting method is enabled forthat tile and cache data (model fingerprints and distance function) iscreated; otherwise, at 732 the fingerprinting method is deemed tooinaccurate and the system “falls back” to use some other method forlocation determination.

This method, illustrated by FIG. 7, provides techniques for forminggeneralized fingerprints that include non-RF related factors such as GPSquality, speed, device type, client ID, OS version, and so forth, bycreating tile specific distance functions from a training dataset andfiltering the training dataset to compute the distance function. Incertain implementations, the method may incrementally update thisdistance function as the training dataset grows. Moreover, this approachalso enables the selection of candidate fingerprints for the distancefunction from all of the crowd sourced data which other methodologies donot fully exploit. In addition, by inferring location from a combinationof the k nearest fingerprints per the distance function, and bypredicting likely error performance (accuracy) from a test dataset, thisapproach also enables the determination of whether the accuracy isacceptable; if not, to fall back to other location methods; and if so,to create cache data for mobile devices to resolve location on thedevice autonomously.

At least a portion of the functionality of the various elements in FIG.2A, FIG. 2B, and FIG. 4 may be performed by other elements in thefigures, or an entity (e.g., processor, web service, server, applicationprogram, computing device, etc.) not shown in the figures. In someimplementations, the operations illustrated in FIG. 3 and FIG. 5 may beimplemented as software instructions encoded on a computer readablemedium, in hardware programmed or designed to perform the operations, orboth. For example, aspects of the disclosure may be implemented as asystem on a chip. Moreover, while no personally identifiable informationis tracked by aspects of the disclosure, implementations have beendescribed with reference to data monitored and/or collected from users.In such implementations, notice is provided to the users of thecollection of the data (e.g., via a dialog box or preference setting)and users are given the opportunity to give or deny consent for themonitoring and/or collection. The consent may take the form of opt-inconsent or opt-out consent.

Of course, the implementations illustrated and described herein as wellas implementations not specifically described herein but within thescope of aspects of the invention constitute exemplary means forcreating models 114 based on the training dataset 106, and exemplarymeans for comparing the accuracy of different modeling algorithms 228and different location inference algorithms 230 based on the aggregatedaccuracy values for the tiles.

FIG. 8 shows an exemplary computing environment in which exampleimplementations and aspects may be implemented. The computing systemenvironment is only one example of a suitable computing environment andis not intended to suggest any limitation as to the scope of use orfunctionality. Numerous other general purpose or special purposecomputing system environments or configurations may be used. Examples ofwell known computing systems, environments, and/or configurations thatmay be suitable for use include, but are not limited to, personalcomputers (PCs), server computers, handheld or laptop devices, mobilecommunications devices, multiprocessor systems, microprocessor-basedsystems, network personal computers, minicomputers, mainframe computers,embedded systems, distributed computing environments that include any ofthe above systems or devices, and the like.

Computer-executable instructions, such as program modules, beingexecuted by a computer may be used. Generally, program modules includeroutines, programs, objects, components, data structures, etc. thatperform particular tasks or implement particular abstract data types.Distributed computing environments may be used where tasks are performedby remote processing devices that are linked through a communicationsnetwork or other data transmission medium. In a distributed computingenvironment, program modules and other data may be located in both localand remote computer storage media including memory storage devices.

With reference to FIG. 8, an exemplary system for implementing aspectsdescribed herein includes a computing device, such as computing device800. In its most basic configuration, computing device 800 typicallyincludes at least one processing unit 802 and memory 804. Depending onthe exact configuration and type of computing device, memory 804 may bevolatile (such as random access memory (RAM)), non-volatile (such asread-only memory (ROM), flash memory, etc.), or some combination of thetwo. This most basic configuration is illustrated in FIG. 8 by dashedline 806.

Computing device 800 may have additional features/functionality. Forexample, computing device 800 may include additional storage (removableand/or non-removable) including, but not limited to, magnetic or opticaldisks or tape. Such additional storage is illustrated in FIG. 8 byremovable storage 808 and non-removable storage 810.

Computing device 800 typically includes a variety of computer readablemedia. Computer readable media can be any available media that can beaccessed by device 800 and includes both volatile and non-volatilemedia, removable and non-removable media.

Computer storage media include volatile and non-volatile, and removableand non-removable media implemented in any method or technology forstorage of information such as computer readable instructions, datastructures, program modules or other data. Memory 804, removable storage808, and non-removable storage 810 are all examples of computer storagemedia. Computer storage media include, but are not limited to, RAM, ROM,electrically erasable program read-only memory (EEPROM), flash memory orother memory technology, CD-ROM, digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed bycomputing device 800. Any such computer storage media may be part ofcomputing device 800.

Computing device 800 may contain communications connection(s) 812 thatallow the device to communicate with other devices. Computing device 800may also have input device(s) 814 such as a keyboard, mouse, pen, voiceinput device, touch input device, etc. Output device(s) 816 such as adisplay, speakers, printer, etc. may also be included. All these devicesare well known in the art and need not be discussed at length here.

It should be understood that the various techniques described herein maybe implemented in connection with hardware or software or, whereappropriate, with a combination of both. Thus, the methods and apparatusof the presently disclosed subject matter, or certain aspects orportions thereof, may take the form of program code (i.e., instructions)embodied in tangible media, such as floppy diskettes, CD-ROMs, harddrives, or any other machine-readable storage medium where, when theprogram code is loaded into and executed by a machine, such as acomputer, the machine becomes an apparatus for practicing the presentlydisclosed subject matter.

Although exemplary implementations may refer to utilizing aspects of thepresently disclosed subject matter in the context of one or morestand-alone computer systems, the subject matter is not so limited, butrather may be implemented in connection with any computing environment,such as a network or distributed computing environment. Still further,aspects of the presently disclosed subject matter may be implemented inor across a plurality of processing chips or devices, and storage maysimilarly be affected across a plurality of devices. Such devices mightinclude personal computers, network servers, and handheld devices, forexample.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

What is claimed:
 1. A method comprising: constructing a training dataset and a test data set by using input data comprising locationinformation and non-location related information; using the trainingdata set and the test data set to generate a first set of performanceanalytics, the first set of performance analytics generated by executinga first location determination procedure; using the training data setand the test data set to generate a second set of performance analytics,the second set of performance analytics generated by executing a secondlocation determination procedure; and determining from at least thefirst set of performance analytics and the second set of performanceanalytics that one of the first location determination procedure or thesecond location determination procedure performs better than the otherof the first location determination procedure or the second locationdetermination procedure.
 2. The method of claim 1, wherein the locationinformation comprises RF-based location information, and wherein thenon-location related information comprises non-RF related factors. 3.The method of claim 2, wherein the non-RF related factors include one ormore of horizontal estimated position error information, device typeinformation, client identification data, device speed, or an operatingsystem.
 4. The method of claim 1, further comprising: generatingpairwise analytics data from the first set of performance analytics andthe second set of performance analytics; and determining that one of thefirst location determination procedure or the second locationdetermination procedure performs better than the other of the firstlocation determination procedure or the second location determinationprocedure based on the pairwise analytics data.
 5. The method of claim4, wherein analyzing the pairwise analytics data comprises analyzing anerror difference in one or more tiles of a grid representing a location.6. The method of claim 5, wherein analyzing the pairwise analytics dataindicates an anomaly.
 7. The method of claim 6, wherein upon detectingthe anomaly, at least one parameter used in at least one of the firstlocation determination procedure or the second location determinationprocedure is changed to a different parameter prior to re-executing theat least one of the first location determination procedure or the secondlocation determination procedure.
 8. The method of claim 6, wherein upondetecting the anomaly, raw data associated with the location informationis debugged.
 9. The method of claim 1, wherein executing the firstlocation determination procedure comprises running a first algorithm,and wherein executing the second location determination procedurecomprises running a second algorithm that is different than the firstalgorithm.
 10. The method of claim 1, wherein the location informationis a crowd-sourced location observation.
 11. A system comprising: amemory associated with a computing device, said memory storing datacomprising one or more location observations and one or more non-RFrelated factors; and a processor programmed to: execute a datasetconstructor to construct a training data set and a test data set usingthe one or more location observations and the one or more non-RF relatedfactors; execute a plurality of experiments to generate comparativeanalytics, wherein executing a first experiment of the plurality ofexperiments comprises: applying a first modeling algorithm and a firstlocation inference algorithm to at least a portion of the test data set;and generating a first set of accuracy analytics from applying the firstmodeling algorithm and the first location inference algorithm to the atleast a portion of the test data set;  and further wherein executing asecond experiment of the plurality of experiments comprises: applying asecond modeling algorithm and a second location inference algorithm tothe at least a portion of the test data set; and generating a second setof accuracy analytics from applying the second modeling algorithm andthe second location inference algorithm to the at least a portion of thetest data set; and execute one or more analytic scripts on each of thefirst and the second set of accuracy analytics to obtain respectivelocation, tile, and world accuracy analytics.
 12. The system of claim11, wherein the processor is further programmed to execute one or moreof a multi-way comparative analysis, a pairwise comparative analysis, adelta comparative analysis, or a correlation comparative analysis on therespective location, tile, and world accuracy analytics.
 13. The systemof claim 11, wherein the one or more non-RF related factors include oneor more of horizontal estimated position error information, device typeinformation, client identification data, device speed, or an operatingsystem.
 14. The system of claim 11, wherein the one or more locationobservations include a crowd-sourced location observation.
 15. A systemcomprising: a memory associated with a computing device, the memorystoring data comprising a training data set and a test data setconstructed by using input data comprising location information andnon-location related information; and a processor programmed to: use thetraining data set and the test data set to generate a first set ofperformance analytics, the first set of performance analytics generatedby executing a first location determination procedure; use the trainingdata set and the test data set to generate a second set of performanceanalytics, the second set of performance analytics generated byexecuting a second location determination procedure; and determine fromat least the first set of performance analytics and the second set ofperformance analytics that one of the first location determinationprocedure or the second location determination procedure performs betterthan the other of the first location determination procedure or thesecond location determination procedure.
 16. The system of claim 15,wherein the location information comprises RF-based locationinformation, and wherein the non-location related information comprisesnon-RF related factors.
 17. The system of claim 16, wherein the non-RFrelated factors include one or more of horizontal estimated positionerror information, device type information, client identification data,device speed, or an operating system.
 18. The system of claim 15,wherein the processor is further programmed to: generate pairwiseanalytics data from the first set of performance analytics and thesecond set of performance analytics; and determine that one of the firstlocation determination procedure or the second location determinationprocedure performs better than the other of the first locationdetermination procedure or the second location determination procedurebased on the pairwise analytics data.
 19. The system of claim 18,wherein analyzing the pairwise analytics data comprises analyzing anerror difference in one or more tiles of a grid representing a location.20. The system of claim 15, wherein the location information is acrowd-sourced location observation.